StaticLLMPipeline: Enable chat test #1117
Conversation
Co-authored-by: Vladimir Zlobin <[email protected]>
ilya-lavrenov
added
category: LLM
| @pytest.mark.precommit | ||
| @pytest.mark.nightly | ||
| def test_chat_generation(model_descr): |
There was a problem hiding this comment.
github-actions
bot
added
category: sampling
|
Any idea why did this happen? |
|
I don't have a solution to |
| questions = [ | ||
| '1+1=', | ||
| 'What is the previous answer?', | ||
| 'Why is the Sun yellow?', | ||
| 'What was my first question?' | ||
| ] | ||
|
|
||
| model_path = get_chat_models_lists()[0][1] | ||
| model_path = get_chat_models_list()[0][1] |
There was a problem hiding this comment.
it's not a model path, it's a model_id. E.g. from CI error we can see:
model_path = WindowsPath('Qwen2-0.5B-Instruct'), device = 'CPU'
which means model is not even converted by Optimum
In other places, it's used like:
pipe = read_model(get_models_list()[0])[4]where read model converts model and created pipe on to top it.
So, the question is - have you run tests locally? do they even magically pass?
There was a problem hiding this comment.
get_chat_models_list() returns List[tuple(model_id, model_path)]
https://github.com/openvinotoolkit/openvino.genai/blob/master/tests/python_tests/ov_genai_test_utils.py#L106
I'd assume get_chat_models_list()[0][1] is the path of the first model, but probably I'm wrong... But at least for get_models_list() it works fine: https://github.com/openvinotoolkit/openvino.genai/blob/master/tests/python_tests/test_llm_pipeline_static.py#L37
Perhaps that problem is that I need to call read_model explicitly at least once, let me try it.
So, the question is - have you run tests locally? do they even magically pass?
I've used my own list of models available on local setup, so yes, I didn't check if this machinery works with default models
No description provided.